Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(blockifier): add NativeSyscallHandler #1157

Conversation

rodrigo-pino
Copy link
Contributor

@rodrigo-pino rodrigo-pino commented Oct 3, 2024

Third time of the PR for the same feature. I am not rebasing from the previous one because git history has been a bit messy and resolving conflicts for 70+ commits is not fun.

This PR comes as a new version for #621 (which in itself was a copy version for #551) but it has been trimmed down a bit.

After this one is merged we can subsequently close the others.


This change is Reviewable

@rodrigo-pino rodrigo-pino changed the base branch from main to native/add-native-execution-engine October 3, 2024 14:02
@rodrigo-pino rodrigo-pino changed the title feat(blockifier): Add Native Syscall Handler feat(blockifier): add NativeSyscallHandler Oct 3, 2024
@rodrigo-pino rodrigo-pino added the native integration Related with the integration of Cairo Native into the Blockifier label Oct 3, 2024
@rodrigo-pino rodrigo-pino force-pushed the rdr/add-native-syscall-handler branch from b15faf6 to 8d8ec18 Compare October 3, 2024 22:04
Copy link

codecov bot commented Oct 3, 2024

Codecov Report

Attention: Patch coverage is 10.24590% with 219 lines in your changes missing coverage. Please review.

Project coverage is 69.23%. Comparing base (2986be6) to head (3b10658).

Files with missing lines Patch % Lines
...blockifier/src/execution/native/syscall_handler.rs 0.00% 219 Missing ⚠️
Additional details and impacted files
@@                          Coverage Diff                           @@
##           native/add-native-execution-engine    #1157      +/-   ##
======================================================================
- Coverage                               70.44%   69.23%   -1.22%     
======================================================================
  Files                                      87       88       +1     
  Lines                                   11512    11756     +244     
  Branches                                11512    11756     +244     
======================================================================
+ Hits                                     8110     8139      +29     
- Misses                                   3033     3248     +215     
  Partials                                  369      369              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Contributor

@meship-starkware meship-starkware left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 1 of 4 files at r1, all commit messages.
Reviewable status: 1 of 4 files reviewed, 2 unresolved discussions (waiting on @rodrigo-pino)


crates/blockifier/src/execution/native/syscall_handler.rs line 110 at r2 (raw file):

    pub fn substract_syscall_gas_cost(
        &mut self,
        remaining_gas: &mut u128,

https://reviewable.io/reviews/starkware-libs/sequencer/621#-O7AR9rr3iLF7ozeCslU
see discussion above

Suggestion:

remaining_gas: &mut u128,
remaining_gas: &mut u64,

crates/blockifier/src/execution/native/utils.rs line 13 at r2 (raw file):

pub fn encode_str_as_felts(msg: &str) -> Vec<Felt> {
    const CHUNK_SIZE: usize = 32;

Can you please explain? Why 32?

Code quote:

const CHUNK_SIZE: usize = 32;

Copy link
Collaborator

@noaov1 noaov1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 1 of 4 files at r1, all commit messages.
Reviewable status: 1 of 4 files reviewed, 5 unresolved discussions (waiting on @meship-starkware and @rodrigo-pino)


crates/blockifier/src/execution/native/syscall_handler.rs line 44 at r2 (raw file):

    pub read_values: Vec<Felt>,
    pub accessed_keys: HashSet<StorageKey, RandomState>,
}

Writing here for the record additional field of the cairo1 vm hint proccessor:
read_only_segments
secp256k\r1_hint_processor
sha256_segment_end_ptr
hints
execution_info_ptr


crates/blockifier/src/execution/native/syscall_handler.rs line 87 at r2 (raw file):

        self.update_remaining_gas(remaining_gas, &call_info);

        let retdata = call_info.execution.retdata.clone();

Suggestion:

        let retdata = call_info.execution.retdata.clone();

        if call_info.execution.failed {
            // In VM it's wrapped into `SyscallExecutionError::SyscallError`.
            return Err(retdata.o.clone());
        }

        self.update_remaining_gas(remaining_gas, &call_info);

crates/blockifier/src/execution/native/syscall_handler.rs line 103 at r2 (raw file):

        // Change the remaining gas value.
        *remaining_gas = u128::from(remaining_gas_u64);
    }

Please move this method outside the Imp

Suggestion:

    pub fn update_remaining_gas(remaining_gas: &mut u128, call_info: &CallInfo) {
        let mut remaining_gas_u64 = u64::try_from(*remaining_gas).expect("Failed to convert gas to u64.");

        update_remaining_gas(&mut remaining_gas_u64, call_info);

        *remaining_gas = u128::from(remaining_gas_u64);
    }

Copy link
Contributor

@meship-starkware meship-starkware left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 1 of 4 files reviewed, 6 unresolved discussions (waiting on @noaov1 and @rodrigo-pino)


crates/blockifier/src/execution/native/utils.rs line 12 at r2 (raw file):

}

pub fn encode_str_as_felts(msg: &str) -> Vec<Felt> {

See discussion
https://reviewable.io/reviews/starkware-libs/sequencer/621#-O5wl5XLBruyQJwstLkp

Copy link
Contributor

@meship-starkware meship-starkware left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 1 of 4 files at r1, 1 of 1 files at r2.
Reviewable status: 3 of 4 files reviewed, 6 unresolved discussions (waiting on @noaov1 and @rodrigo-pino)

@rodrigo-pino rodrigo-pino force-pushed the native/add-native-execution-engine branch from 3d1b52d to 0b4b3bf Compare October 6, 2024 15:12
Copy link
Contributor Author

@rodrigo-pino rodrigo-pino left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 3 of 4 files reviewed, 6 unresolved discussions (waiting on @meship-starkware and @noaov1)


crates/blockifier/src/execution/native/syscall_handler.rs line 87 at r2 (raw file):

        self.update_remaining_gas(remaining_gas, &call_info);

        let retdata = call_info.execution.retdata.clone();

👍


crates/blockifier/src/execution/native/syscall_handler.rs line 103 at r2 (raw file):

Previously, noaov1 (Noa Oved) wrote…

Please move this method outside the Imp

I don't know why this were public execute_inner_call, update_remaining_gas and substract_syscall_gas_cost. I've reverted them back to private and used #[allow(dead_code)] for the time being.

I can also removed them but almost all syscalls will be using them in the future which means a bit slowness when starting those.

Is in the impl because it was meant as an internal part of NativeSyscallHandler private to everyone else. If you still think is best out I'll move it out


crates/blockifier/src/execution/native/syscall_handler.rs line 110 at r2 (raw file):

Previously, meship-starkware (Meshi Peled) wrote…

https://reviewable.io/reviews/starkware-libs/sequencer/621#-O7AR9rr3iLF7ozeCslU
see discussion above

Native treats gas as a u128 hence we have to build our wrappers around it to communicate with blockifiers u64. We can ask for a u64 change on their part and re-update this part.


crates/blockifier/src/execution/native/utils.rs line 12 at r2 (raw file):

Previously, meship-starkware (Meshi Peled) wrote…

See discussion
https://reviewable.io/reviews/starkware-libs/sequencer/621#-O5wl5XLBruyQJwstLkp

This functions are required to handle user created errors (either panics! or Err results) or inner-call error propagation.

If we get an error message which fits in a Felt then it behavesthe same as the VM.

If the error is bigger, meaning it spans several Felt, we take the bytes of each felt, join them and decode them as utf8 if possible. If not utf8-decodable, then we just try to decode individual Felts.

This method is only used after Native returned control and there was an error. The error returned by Native won't be in String form but in a Vec<Felt> hence the need to decode it.


crates/blockifier/src/execution/native/utils.rs line 13 at r2 (raw file):

Previously, meship-starkware (Meshi Peled) wrote…

Can you please explain? Why 32?

We are converting a string back into a Felt. We then created small arrays of 32 bytes, fill the first 31 bytes with the string encoding information and make them Felts.

@rodrigo-pino rodrigo-pino force-pushed the rdr/add-native-syscall-handler branch 2 times, most recently from 667ca73 to 68e2ecb Compare October 6, 2024 16:30
Copy link
Collaborator

@noaov1 noaov1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 3 of 17 files at r3, 1 of 1 files at r6, all commit messages.
Reviewable status: 4 of 17 files reviewed, 7 unresolved discussions (waiting on @meship-starkware and @rodrigo-pino)


.github/workflows/blockifier_ci.yml line 44 at r7 (raw file):

jobs:
  featureless-build:
    runs-on: ubuntu-22.04

Why was this change needed?
@alon-dotan-starkware

Code quote:

    runs-on: ubuntu-22.04

.github/workflows/main.yml line 22 at r7 (raw file):

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: ${{ github.event_name == 'pull_request' }}

Why was this removed?

Code quote:

# On PR events, cancel existing CI runs on this same PR for this workflow.
concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: ${{ github.event_name == 'pull_request' }}

crates/blockifier/src/execution/native/syscall_handler.rs line 103 at r2 (raw file):

Previously, rodrigo-pino (Rodrigo) wrote…

I don't know why this were public execute_inner_call, update_remaining_gas and substract_syscall_gas_cost. I've reverted them back to private and used #[allow(dead_code)] for the time being.

I can also removed them but almost all syscalls will be using them in the future which means a bit slowness when starting those.

Is in the impl because it was meant as an internal part of NativeSyscallHandler private to everyone else. If you still think is best out I'll move it out

RE the private: great!
I thought that as it does not use self it is better to move it out of the impl block.

@rodrigo-pino rodrigo-pino force-pushed the rdr/add-native-syscall-handler branch 2 times, most recently from 19f2910 to 425acc7 Compare October 6, 2024 20:09
Copy link
Collaborator

@noaov1 noaov1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 1 of 4 files at r9, 1 of 1 files at r11, 1 of 1 files at r13, 1 of 1 files at r14, 1 of 1 files at r16, 10 of 10 files at r17, all commit messages.
Reviewable status: 15 of 17 files reviewed, 8 unresolved discussions (waiting on @meship-starkware and @rodrigo-pino)


crates/blockifier/src/execution/native/syscall_handler.rs line 112 at r17 (raw file):

                    .expect("Failed to parse OUT_OF_GAS_ERROR hex string"),
            ]);
        }

Where do we set the failure flag to 1 in this case? (and in any case of an error returned from a syscall)

Code quote:

            return Err(vec![
                Felt::from_hex(OUT_OF_GAS_ERROR)
                    .expect("Failed to parse OUT_OF_GAS_ERROR hex string"),
            ]);
        }

crates/blockifier/src/execution/native/syscall_handler.rs line 320 at r17 (raw file):

fn native_update_remaining_gas(remaining_gas: &mut u128, call_info: &CallInfo) {
    // Create a new variable with converted type.
    let mut remaining_gas_u64 = u64::try_from(*remaining_gas).unwrap();

Suggestion:

let mut remaining_gas_u64 = u64::try_from(*remaining_gas).expect("Failed to convert gas to u64.");

crates/blockifier/src/execution/native/utils.rs line 13 at r2 (raw file):

Previously, rodrigo-pino (Rodrigo) wrote…

We are converting a string back into a Felt. We then created small arrays of 32 bytes, fill the first 31 bytes with the string encoding information and make them Felts.

Why 32 exactly?

Copy link
Contributor Author

@rodrigo-pino rodrigo-pino left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 14 of 17 files reviewed, 8 unresolved discussions (waiting on @meship-starkware and @noaov1)


crates/blockifier/src/execution/native/syscall_handler.rs line 103 at r2 (raw file):

Previously, noaov1 (Noa Oved) wrote…

RE the private: great!
I thought that as it does not use self it is better to move it out of the impl block.

yes, it makes sense!


crates/blockifier/src/execution/native/syscall_handler.rs line 112 at r17 (raw file):

Previously, noaov1 (Noa Oved) wrote…

Where do we set the failure flag to 1 in this case? (and in any case of an error returned from a syscall)

When Native invokes a syscall and passes control to us, it expects a a result a SyscallResult<OkType, ErrType>. If we return an Err in the syscall, in this specific context by calling this function, it will propagate back to Native. The Native Contract should then propagate the error back and return it as well as setting the failure flag to 1


crates/blockifier/src/execution/native/syscall_handler.rs line 320 at r17 (raw file):

fn native_update_remaining_gas(remaining_gas: &mut u128, call_info: &CallInfo) {
    // Create a new variable with converted type.
    let mut remaining_gas_u64 = u64::try_from(*remaining_gas).unwrap();

👍


crates/blockifier/src/execution/native/utils.rs line 13 at r2 (raw file):

Previously, noaov1 (Noa Oved) wrote…

Why 32 exactly?

Ok, so the issue to solve is we have a long string we want to encode as Felts. We know how much info fits in a Felt (31 Bytes).

The function splits the string in chunks of 31 Bytes, copies them to a static array of size 32 Bytes which is then used to create the new felt with Felt::from_bytes_be(...). This function requires the static array size to be 32 (maybe this is what you were asking)

During the code we are either dealing with CHUNK_SIZE - 1 or CHUNK_SIZE.


.github/workflows/main.yml line 22 at r7 (raw file):

Previously, noaov1 (Noa Oved) wrote…

Why was this removed?

Unintentional deletion.

Copy link
Contributor

@meship-starkware meship-starkware left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 2 of 4 files at r9, 1 of 1 files at r10, 1 of 1 files at r11, 1 of 1 files at r13, 1 of 1 files at r14, 1 of 1 files at r16, 10 of 10 files at r17, 1 of 1 files at r18, all commit messages.
Reviewable status: all files reviewed, 5 unresolved discussions (waiting on @noaov1 and @rodrigo-pino)

@rodrigo-pino rodrigo-pino force-pushed the native/add-native-execution-engine branch 2 times, most recently from ceeeaa4 to 9a9e4bb Compare October 7, 2024 15:39
Copy link
Collaborator

@noaov1 noaov1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 1 of 1 files at r18, all commit messages.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @rodrigo-pino)


crates/blockifier/src/execution/native/utils.rs line 13 at r2 (raw file):

Previously, rodrigo-pino (Rodrigo) wrote…

Ok, so the issue to solve is we have a long string we want to encode as Felts. We know how much info fits in a Felt (31 Bytes).

The function splits the string in chunks of 31 Bytes, copies them to a static array of size 32 Bytes which is then used to create the new felt with Felt::from_bytes_be(...). This function requires the static array size to be 32 (maybe this is what you were asking)

During the code we are either dealing with CHUNK_SIZE - 1 or CHUNK_SIZE.

Got it. Thanks!
Why not do it as we encode the constants error (e.g. OUT_OF_GAS_ERROR):

Code snippet:

let bytes = msg.as_bytes();
let error_felt = BigUint::from_bytes_be(bytes)
let error_hex = format!("0x{:x}", error_felt)
 Felt::from_hex(error_hex)

Copy link
Collaborator

@noaov1 noaov1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: all files reviewed, 2 unresolved discussions (waiting on @rodrigo-pino)

rodrigo-pino and others added 15 commits October 7, 2024 12:42
List of files restored:
.github/workflows/blockifier_ci.yml
.github/workflows/blockifier_compiled_cairo.yml
.github/workflows/clean_stale_prs.yml
.github/workflows/lock_closed_prs.yml
.github/workflows/main.yml
.github/workflows/merge_paths_ci.yml
.github/workflows/papyrus/helm-install.yml
.github/workflows/papyrus_benchmark.yaml
.github/workflows/papyrus_ci.yml
.github/workflows/verify-deps.yml
@rodrigo-pino rodrigo-pino force-pushed the rdr/add-native-syscall-handler branch from e6d93b1 to 3b10658 Compare October 7, 2024 16:45
Copy link
Contributor Author

@rodrigo-pino rodrigo-pino left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 16 of 18 files reviewed, 2 unresolved discussions (waiting on @meship-starkware and @noaov1)


crates/blockifier/src/execution/native/utils.rs line 13 at r2 (raw file):

Previously, noaov1 (Noa Oved) wrote…

Got it. Thanks!
Why not do it as we encode the constants error (e.g. OUT_OF_GAS_ERROR):

In this case we are just handling case where msg cannot have more than 31 bytes of information, otherwise it might not fit in a felt.

I remember we took that decision because there are errors given by the blockifier which exceed the 31 byte size, and there was no way of propagating all the information back through Native the same way you do in the VM with the Blockifier.

For example, the following error, there is no way to propagate back with Native without encoding it as Felts:

#[error("Unauthorized syscall {syscall_name} in execution mode {execution_mode}.")]
InvalidSyscallInExecutionMode { syscall_name: String, execution_mode: ExecutionMode },

@noaov1 noaov1 deleted the branch native/add-native-execution-engine October 7, 2024 19:56
@noaov1 noaov1 closed this Oct 7, 2024
@github-actions github-actions bot locked and limited conversation to collaborators Oct 10, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
native integration Related with the integration of Cairo Native into the Blockifier
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants